Advanced Lane Finding Project

The goals / steps of this project are the following:

1. Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
2. Apply a distortion correction to raw images.
3. Use color transforms, gradients, etc., to create a thresholded binary image.
4. Apply a perspective transform to rectify binary image ("birds-eye view").
5. Detect lane pixels and fit to find the lane boundary.
6. Determine the curvature of the lane and vehicle position with respect to center.
7. Warp the detected lane boundaries back onto the original image.
8. Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

step - 1

1. Creating the 3d objPoints and imgPoints.
2. using findChessboardCorners to detect the corners in the image
3. Drawing the detected corners on the image.

# Camera Calibration matrix and distortion coefiicients 
import numpy as np
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import glob
%matplotlib inline

# Arrays to store object points and image points from all the images
objPoints = []
imgPoints = []
 
# prepare object points 
objP = np.zeros((6*9,3), np.float32)
objP[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2) 

imgContainer = glob.glob('camera_cal/*.jpg')

## finding and drawing the image corners on the image
for i, imgName in enumerate(imgContainer):
    image = cv2.imread(imgName)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    ret, corners = cv2.findChessboardCorners(gray, (9,6), None)
    
    if ret == True:
        imgPoints.append(corners)
        objPoints.append(objP)
        f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
        img = cv2.drawChessboardCorners(image, (9,6), corners, ret)
        axs1.set_title("Original image")
        axs1.imshow(image)
        axs2.set_title("Corners image")
        axs2.imshow(img)



print("Completed")

Completed

step - 2

Camera Calibration

1. I have used OpenCV functions findChessboardCorners and calibrateCamera for the image calibration.
2. Input consited of chessboar patterns with different angles, color etc.
3. Detecting chessboard corners
4. Drawing the chessboard corners using the openCV functions.
5. Undistort function to undistort the image.

## calibrating the camera and undistorting the image
def cal_undistort(img):
    # Use cv2.calibrateCamera() and cv2.undistort()
    img_size = (img.shape[1], img.shape[0])
    ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objPoints, imgPoints, img_size, None, None)
    dst = cv2.undistort(img, mtx, dist, None, mtx)
    return dst

print("Completed")

Completed

## applying the undistorting to each individual images
for image in imgContainer:
    img = cv2.imread(image)
    undistort = cal_undistort(img)
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(img)
    axs2.set_title("Undistorted image")
    axs2.imshow(undistort)

print("Completed")

Completed

step - 3 Creating bird's eye view

Perspective transform

1. A perspective transform maps the points in a given image to different, desired, image points with a new perspective
2. Select 4 source points according to a given image and its content
3. Select 4 destination points like rectangle
4. Now we do a perspective transfrom to get a bird's eye view

# defining source points and destination points
imgHolder = glob.glob('test_images/*.jpg')
testImage = cv2.imread('test_images/test2.jpg')
plt.imshow(testImage)
plt.plot(300,700, '.') #- left bottom
plt.plot(560,460, '.') #- top left
plt.plot(700,460,'.')  #- top right
plt.plot(1150,700,'.') #- right bottom
#srcPoints


def warp(img):
    img_size = (img.shape[1], img.shape[0])
    # creating the source points
    srcPoints = np.float32(
                [[300,700],
                 [560,460],
                 [700,460],
                 [1150,700]])
    dstPoints = np.float32(
                [[400,700],
                 [400,100],
                 [980,100],
                 [980,700]])
    M = cv2.getPerspectiveTransform(srcPoints, dstPoints)
    Minv = cv2.getPerspectiveTransform(dstPoints, srcPoints)
    warped = cv2.warpPerspective(img, M, img_size, flags=cv2.INTER_LINEAR)
    
    return warped, M,  Minv

print("Completed")

Completed

for img in imgHolder:
    image = cv2.imread(img)
    warped, M, Minv = warp(image)
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(warped)
    
print("Completed")

Completed

step - 4 Gradient thresholding

Threshold

1. Here i have used sobel threshold, magnitude threshold and direction threshold on the input images
2. Sobel threshold gives better output with orient as x, and after experimenting the better threshold was 20,100.
3. Similary for magnitude threshold - 3,100
4. Similary for direction threshold - 0.7, 1.3

Much more experimentation can be done using different kernel sizes, differnt thresholds.

## applying sobel operator on the images
def sobel(img, orient='x', thresh_min = 0, thresh_max= 255):
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    if orient == 'x':
        abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 1, 0))
    if orient == 'y':
        abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 0, 1))
    scaled_sobel = np.uint8(255*abs_sobel/np.max(abs_sobel))
    thresh_min = 20
    thresh_max = 100
    sxbinary = np.zeros_like(scaled_sobel)
    sxbinary[(scaled_sobel >= thresh_min) & (scaled_sobel <= thresh_max)] = 1
    #plt.imshow(sxbinary, cmap='gray')
    return sxbinary
print("Completed")

Completed

for img in imgHolder:
    image = cv2.imread(img)
    outSobel = sobel(image, orient='x', thresh_min=20, thresh_max=100)
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(outSobel, cmap='gray')

print("Completed")

Completed

## Applying the magnitude threshold
def mag_thresh(img, sobel_kernel=3, mag_thresh=(0, 255)):
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
    sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
    gradmag = np.sqrt(sobelx**2 + sobely**2)
    # Rescale to 8 bit
    scale_factor = np.max(gradmag)/255 
    gradmag = (gradmag/scale_factor).astype(np.uint8) 
    # Create a binary image of ones where threshold is met, zeros otherwise
    binary_output = np.zeros_like(gradmag)
    binary_output[(gradmag >= mag_thresh[0]) & (gradmag <= mag_thresh[1])] = 1
    return binary_output

print("Completed")

Completed

for img in imgHolder:
    image = cv2.imread(img)
    magSobel = mag_thresh(image, sobel_kernel=31, mag_thresh=(40, 100))
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(magSobel, cmap='gray')

print("Completed")

Completed

## Applying direction threshold
def dir_threshold(img, sobel_kernel=3, thresh=(0, np.pi/2)):
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    # Calculate the x and y gradients
    sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
    sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
    # Take the absolute value of the gradient direction, 
    # apply a threshold, and create a binary image result
    absgraddir = np.arctan2(np.absolute(sobely), np.absolute(sobelx))
    binary_output =  np.zeros_like(absgraddir)
    binary_output[(absgraddir >= thresh[0]) & (absgraddir <= thresh[1])] = 1
    return binary_output

print("Completed")

Completed

for img in imgHolder:
    image = cv2.imread(img)
    dirSobel= dir_threshold(image, sobel_kernel=15, thresh=(0.7, 1.3))
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(dirSobel, cmap='gray')

print("Completed")

Completed

step - 5 combining threshold

Combined

1. Here i have tried to combined 2 differnt thresholds and then outputting the image.
2. Also i have used the same threshold, kernel sizes as used above.

def combined_threshold(img):
    image = cv2.imread(img)
    gradx = sobel(image, orient='x', thresh_min= 30, thresh_max = 100)
    grady = sobel(image, orient='y', thresh_min= 30, thresh_max = 100)
    mag_binary = mag_thresh(image, sobel_kernel=15, mag_thresh=(40, 100))
    dir_binary = dir_threshold(image, sobel_kernel=15, thresh=(0.7, 1.3))
    combined = np.zeros_like(dir_binary)
    combined[((gradx == 1) & (grady == 1)) | ((mag_binary == 1) & (dir_binary == 1))] = 1
    
    return combined

for img in imgHolder:
    combined = combined_threshold(img)
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(combined, cmap='gray')

print("Completed")

Completed

step - 6 Color thresholds

Color threshold explanation

This was the most interesting part of the project. I tried experimenting with different color spaces and different channels with different thresholds.

1. Initially i started with a RGB color image and save its 3 channels in different variables.
2. Applied threshold and then i was able to find that R and G gives better output with most of the images but blue fails to detect yellow lines
3. Next i used HLS color space.
4. Used 3 differnt channel variables for the 3 different channels.
5. H,L, S almost all 3 of them gives better results.
6. L and S were able to detect the lane lines in a better way.

But these outputs can be further improved and experimented with differnt thresholds, differnt cmbinations of thresholds.

def rgb_thresh(img):
    R = image[:,:,0]
    G = image[:,:,1]
    thresh = (200, 255)
    binaryR = np.zeros_like(R)
    binaryR[(R > thresh[0]) & (R <= thresh[1])] = 1
    binaryG = np.zeros_like(G)
    binaryG[(G > thresh[0]) & (G <= thresh[1])] = 1
    return binaryR, binaryG

def hls_thresh(img):
    hls = cv2.cvtColor(image, cv2.COLOR_BGR2HLS)
    H = hls[:,:,0]
    L = hls[:,:,1]
    S = hls[:,:,2]    
    thresh = (90, 255)
    binaryS = np.zeros_like(S)
    binaryS[(S > thresh[0]) & (S <= thresh[1])] = 1
    threshHLS = (15, 100)
    binaryH = np.zeros_like(H)
    binaryH[(H > threshHLS[0]) & (H <= threshHLS[1])] = 1
    return binaryS, binaryH

for img in imgHolder:
    image = cv2.imread(img)
    R, G = rgb_thresh(image)
    binaryS, binaryH = hls_thresh(image)
    f, (axs0, axs1, axs2, axs3, axs4) = plt.subplots(1, 5, figsize=(15,11))
    axs0.set_title("Original Image")
    axs0.imshow(image)
    axs1.set_title("RGB R-channel image")
    axs1.imshow(R, cmap = 'gray')
    axs2.set_title("RGB G-channel image")
    axs2.imshow(G, cmap='gray')
    axs3.set_title("HLS H-channel image")
    axs3.imshow(binaryS, cmap = 'gray')
    axs4.set_title("HLS S-channel image")
    axs4.imshow(binaryH, cmap = 'gray')
    
print("Completed")

Completed

### step - 7 pipeline

Pipeline - It does all the above steps, I have only passed the images from the folder to a series of steps(Mentioned above) within the pipeline module.

def pipeline(img, s_thresh=(170, 255), sx_thresh=(20, 100)):
    img = np.copy(img)
    # Convert to HLS color space and separate the V channel
    hls = cv2.cvtColor(img, cv2.COLOR_BGR2HLS).astype(np.float)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    l_channel = hls[:,:,1]
    s_channel = hls[:,:,2]
    # Sobel x
    sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0) # Take the derivative in x
    abs_sobelx = np.absolute(sobelx) # Absolute x derivative to accentuate lines away from horizontal
    scaled_sobel = np.uint8(255*abs_sobelx/np.max(abs_sobelx))
    
    # Threshold x gradient
    sxbinary = np.zeros_like(scaled_sobel)
    sxbinary[(scaled_sobel >= sx_thresh[0]) & (scaled_sobel <= sx_thresh[1])] = 1
    
    # Threshold color channel
    s_binary = np.zeros_like(s_channel)
    s_binary[(s_channel >= s_thresh[0]) & (s_channel <= s_thresh[1])] = 1
    # Stack each channel
    # Combine the two binary thresholds
    combined_binary = np.zeros_like(sxbinary)
    combined_binary[(s_binary == 1) | (sxbinary == 1)] = 1
    # Note color_binary[:, :, 0] is all 0s, effectively an all black image. It might
    # be beneficial to replace this channel with something else.
    color_binary = np.dstack(( np.zeros_like(sxbinary), sxbinary, s_binary)) * 255
    return combined_binary


for img in imgHolder:
    image = cv2.imread(img)
    result = pipeline(image)
    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(result, cmap='gray')

print("Completed")

Completed

step - 8. Line Finding

Describe how (and identify where in your code) you identified lane-line pixels and fit their positions with a polynomial?

Solution -

1. The functions sliding_window and advanceLane, which helps in identifying lane lines and fit a second order polynomial to both right and left lane lines.
2. The first step - Computes a histogram of the bottom half of the image and finds the bottom-most x position (or "base") of the left and right lane lines.
3. With this histogram I am adding up the pixel values along each column in the image. In my thresholded binary image, pixels are either 0 or 1, so the two most prominent peaks in this histogram will be good indicators of the x-position of the base of the lane lines. I can use that as a starting point for where to search for the lines. From that point, I can use a sliding window, placed around the line centers, to find and follow the lines up to the top of the frame.
4. Next doing the search along the margin aorund the previous line position detected in the sliding window.
5. And then doing the visualization for both of them.

inputImage = cv2.imread('./test_images/test3.jpg')
testWarp, M, Minv = warp(inputImage)
testWarpedImg = pipeline(testWarp)
plt.imshow(testWarpedImg, cmap='gray')
print("Completed")

Completed

import numpy as np
histogram = np.sum(testWarpedImg[testWarpedImg.shape[0]//2:,:], axis=0)
plt.plot(histogram)
print("Completed")

Completed

### Sliding window 
def sliding_window(binary_warped):
    histogram = np.sum(binary_warped[binary_warped.shape[0]//2:,:], axis=0)
    # Create an output image to draw on and  visualize the result
    out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
    # Find the peak of the left and right halves of the histogram
    # These will be the starting point for the left and right lines
    midpoint = np.int(histogram.shape[0]/2)
    leftx_base = np.argmax(histogram[:midpoint])
    rightx_base = np.argmax(histogram[midpoint:]) + midpoint

    # Choose the number of sliding windows
    nwindows = 9
    # Set height of windows
    window_height = np.int(binary_warped.shape[0]/nwindows)
    # Identify the x and y positions of all nonzero pixels in the image
    nonzero = binary_warped.nonzero()
    nonzeroy = np.array(nonzero[0])
    nonzerox = np.array(nonzero[1])
    # Current positions to be updated for each window
    leftx_current = leftx_base
    rightx_current = rightx_base
    # Set the width of the windows +/- margin
    margin = 80
    # Set minimum number of pixels found to recenter window
    minpix = 30
    # Create empty lists to receive left and right lane pixel indices
    left_lane_inds = []
    right_lane_inds = []

    # Step through the windows one by one
    for window in range(nwindows):
        # Identify window boundaries in x and y (and right and left)
        win_y_low = binary_warped.shape[0] - (window+1)*window_height
        win_y_high = binary_warped.shape[0] - window*window_height
        win_xleft_low = leftx_current - margin
        win_xleft_high = leftx_current + margin
        win_xright_low = rightx_current - margin
        win_xright_high = rightx_current + margin
        # Draw the windows on the visualization image
        cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),
        (0,255,0), 2) 
        cv2.rectangle(out_img,(win_xright_low,win_y_low),(win_xright_high,win_y_high),
        (0,255,0), 2) 
        # Identify the nonzero pixels in x and y within the window
        good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & 
        (nonzerox >= win_xleft_low) &  (nonzerox < win_xleft_high)).nonzero()[0]
        good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) & 
        (nonzerox >= win_xright_low) &  (nonzerox < win_xright_high)).nonzero()[0]
        # Append these indices to the lists
        left_lane_inds.append(good_left_inds)
        right_lane_inds.append(good_right_inds)
        # If you found > minpix pixels, recenter next window on their mean position
        if len(good_left_inds) > minpix:
            leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
        if len(good_right_inds) > minpix:        
            rightx_current = np.int(np.mean(nonzerox[good_right_inds]))

    # Concatenate the arrays of indices
    left_lane_inds = np.concatenate(left_lane_inds)
    right_lane_inds = np.concatenate(right_lane_inds)

    # Extract left and right line pixel positions
    leftx = nonzerox[left_lane_inds]
    lefty = nonzeroy[left_lane_inds] 
    rightx = nonzerox[right_lane_inds]
    righty = nonzeroy[right_lane_inds] 

    # Fit a second order polynomial to each
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)
    
    return left_fit, right_fit, left_lane_inds, right_lane_inds, out_img, nonzeroy, nonzerox 

print("Completed")

Completed

### Visualization
binary_warped = testWarpedImg
left_fit, right_fit, left_lane_inds, right_lane_inds, out_img, nonzeroy, nonzerox = sliding_window(binary_warped)
ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]

out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]
plt.imshow(out_img)
plt.plot(left_fitx, ploty, color='yellow')
plt.plot(right_fitx, ploty, color='yellow')
plt.xlim(0, 1280)
plt.ylim(720, 0)
print("Completed")

Completed

def advanceLane(binary_warped, left_fit, right_fit):
    nonzero = binary_warped.nonzero()
    nonzeroy = np.array(nonzero[0])
    nonzerox = np.array(nonzero[1])
    margin = 100
    left_lane_inds = ((nonzerox > (left_fit[0]*(nonzeroy**2) + left_fit[1]*nonzeroy + 
    left_fit[2] - margin)) & (nonzerox < (left_fit[0]*(nonzeroy**2) + 
    left_fit[1]*nonzeroy + left_fit[2] + margin))) 

    right_lane_inds = ((nonzerox > (right_fit[0]*(nonzeroy**2) + right_fit[1]*nonzeroy + 
    right_fit[2] - margin)) & (nonzerox < (right_fit[0]*(nonzeroy**2) + 
    right_fit[1]*nonzeroy + right_fit[2] + margin)))  

    # Again, extract left and right line pixel positions
    leftx = nonzerox[left_lane_inds]
    lefty = nonzeroy[left_lane_inds] 
    rightx = nonzerox[right_lane_inds]
    righty = nonzeroy[right_lane_inds]
    # Fit a second order polynomial to each
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)
    # Generate x and y values for plotting
    ploty = np.linspace(0, binary_warped.shape[0]-1, binary_warped.shape[0] )
    left_fitx = left_fit[0]*ploty**2 + left_fit[1]*ploty + left_fit[2]
    right_fitx = right_fit[0]*ploty**2 + right_fit[1]*ploty + right_fit[2]
    
    return left_fitx, right_fitx, left_lane_inds, right_lane_inds, ploty

print("Completed")

Completed

### Visualize
margin = 100
left_fitx, right_fitx, left_lane_inds2, right_lane_inds2, ploty = advanceLane(binary_warped, left_fit, right_fit)
# Create an image to draw on and an image to show the selection window
out_img = np.dstack((binary_warped, binary_warped, binary_warped))*255
window_img = np.zeros_like(out_img)
# Color in left and right line pixels
out_img[nonzeroy[left_lane_inds2], nonzerox[left_lane_inds2]] = [255, 0, 0]
out_img[nonzeroy[right_lane_inds2], nonzerox[right_lane_inds2]] = [0, 0, 255]

# Generate a polygon to illustrate the search window area
# And recast the x and y points into usable format for cv2.fillPoly()
left_line_window1 = np.array([np.transpose(np.vstack([left_fitx-margin, ploty]))])
left_line_window2 = np.array([np.flipud(np.transpose(np.vstack([left_fitx+margin, 
                              ploty])))])
left_line_pts = np.hstack((left_line_window1, left_line_window2))
right_line_window1 = np.array([np.transpose(np.vstack([right_fitx-margin, ploty]))])
right_line_window2 = np.array([np.flipud(np.transpose(np.vstack([right_fitx+margin, 
                              ploty])))])
right_line_pts = np.hstack((right_line_window1, right_line_window2))

# Draw the lane onto the warped blank image
cv2.fillPoly(window_img, np.int_([left_line_pts]), (0,255, 0))
cv2.fillPoly(window_img, np.int_([right_line_pts]), (0,255, 0))
result = cv2.addWeighted(out_img, 1, window_img, 0.3, 0)
plt.imshow(result)
plt.plot(left_fitx, ploty, color='yellow')
plt.plot(right_fitx, ploty, color='yellow')
plt.xlim(0, 1280)
plt.ylim(720, 0)
print("Completed")

Completed

Describe how (and identify where in your code) you calculated the radius of curvature of the lane and the position of the vehicle with respect to center.

Solutions -

1. This is where i have computed the curvature radius and the position of the vehicle. The radius of curvature is computed in the calc_curv_rad_and_center_dist method. The pixel values of the lane are scaled into meters using the scaling factors defined as follows:

ym_per_pix = 30/720 # meters per pixel in y dimension xm_per_pix = 3.7/700 # meteres per pixel in x dimension

I have used these values to compute the polynomial coefficients in meters and then used the formula given in the lectures to compute the radius of curvature.

Since the assumption is based on the fact that camera is placed on the center and then checks how far the midpoint of the two lanes is from the center of the image.

car_position = binary_warped.shape[1]/2
lane_center_position = (left_curverad + right_curverad) /2
center_dist = (car_position - lane_center_position) * xm_per_pix

### get radius of curvature
def calc_curv_rad_and_center_dist(left_fit, right_fit, left_lane_inds, right_lane_inds, ploty):
    ploty = np.linspace(0, 719, num=720)# to cover same y-range as image
    quadratic_coeff = 3e-4 
    # For each y position generate random x position within +/-50 pix
    # of the line base position in each case (x=200 for left, and x=900 for right)
    leftx = np.array([200 + (y**2)*quadratic_coeff + np.random.randint(-50, high=51) 
                                  for y in ploty])
    rightx = np.array([900 + (y**2)*quadratic_coeff + np.random.randint(-50, high=51) 
                                    for y in ploty])

    leftx = leftx[::-1]  # Reverse to match top-to-bottom in y
    rightx = rightx[::-1]  # Reverse to match top-to-bottom in y

    # Define y-value where we want radius of curvature
    # I'll choose the maximum y-value, corresponding to the bottom of the image
    y_eval = np.max(ploty)
    left_curverad = ((1 + (2*left_fit[0]*y_eval + left_fit[1])**2)**1.5) / np.absolute(2*left_fit[0])
    right_curverad = ((1 + (2*right_fit[0]*y_eval + right_fit[1])**2)**1.5) / np.absolute(2*right_fit[0])
    # Define conversions in x and y from pixels space to meters
    ym_per_pix = 30/720 # meters per pixel in y dimension
    xm_per_pix = 3.7/700 # meters per pixel in x dimension

    # Fit new polynomials to x,y in world space
    left_fit_cr = np.polyfit(ploty*ym_per_pix, leftx*xm_per_pix, 2)
    right_fit_cr = np.polyfit(ploty*ym_per_pix, rightx*xm_per_pix, 2)
    # Calculate the new radii of curvature
    left_curverad = ((1 + (2*left_fit_cr[0]*y_eval*ym_per_pix + left_fit_cr[1])**2)**1.5) / np.absolute(2*left_fit_cr[0])
    right_curverad = ((1 + (2*right_fit_cr[0]*y_eval*ym_per_pix + right_fit_cr[1])**2)**1.5) / np.absolute(2*right_fit_cr[0])
    # Now our radius of curvature is in meters
    car_position = binary_warped.shape[1]/2
    lane_center_position = (left_curverad + right_curverad) /2
    center_dist = (car_position - lane_center_position) * xm_per_pix
    #print(left_curverad, 'm', right_curverad, 'm')

    return left_curverad, right_curverad, center_dist

print(calc_curv_rad_and_center_dist(left_fit, right_fit, left_lane_inds, right_lane_inds, ploty))    
print("Completed")

(524.84557015972655, 508.58609080625621, 0.65164489601847431)
Completed

# Create an image to draw the lines on
original_image = inputImage
binary_warped = testWarpedImg
warp_zero = np.zeros_like(binary_warped).astype(np.uint8)
color_warp = np.dstack((warp_zero, warp_zero, warp_zero))

# Recast the x and y points into usable format for cv2.fillPoly()
pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
pts = np.hstack((pts_left, pts_right))

# Draw the lane onto the warped blank image
cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))

# Warp the blank back to original image space using inverse perspective matrix (Minv)
warped, M, Minv = warp(testWarpedImg)

newwarp = cv2.warpPerspective(color_warp, Minv, (testImage.shape[1], testImage.shape[0])) 
# Combine the result with the original image
result = cv2.addWeighted(original_image, 1, newwarp, 0.3, 0)
plt.imshow(result)

<matplotlib.image.AxesImage at 0x12bfa0668>

Applying the pipeline to all the images

Provide an example image of your result plotted back down onto the road such that the lane area is identified clearly.

1. The above image shows clearly that lane is detected properly
2. Drew the lane on the original image.
3. The step below i have drawn the calculation for curvature and radius on top of the imagepipeline image.

def imagePipeline(img):
    img = cal_undistort(img)
    image, M, Minv = warp(img)
    binary_warped = pipeline(image)
    left_fit, right_fit, left_lane_inds, right_lane_inds, out_img, nonzeroy, nonzerox = sliding_window(binary_warped)
    left_fitx, right_fitx, left_lane_inds2, right_lane_inds2, ploty = advanceLane(binary_warped, left_fit, right_fit)
    
    left_curverad, right_curverad, center_dist = calc_curv_rad_and_center_dist(left_fit, right_fit, left_lane_inds, right_lane_inds, ploty)
    
    warp_zero = np.zeros_like(binary_warped).astype(np.uint8)
    color_warp = np.dstack((warp_zero, warp_zero, warp_zero))

    # Recast the x and y points into usable format for cv2.fillPoly()
    pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
    pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
    pts = np.hstack((pts_left, pts_right))

    # Draw the lane onto the warped blank image
    cv2.fillPoly(color_warp, np.int_([pts]), (0,255, 0))

    # Warp the blank back to original image space using inverse perspective matrix (Minv)
    warped, M, Minv = warp(binary_warped)

    newwarp = cv2.warpPerspective(color_warp, Minv, (testImage.shape[1], testImage.shape[0])) 
    # Combine the result with the original image
    result = cv2.addWeighted(img, 1, newwarp, 0.3, 0)
    
    return result, left_curverad, right_curverad, center_dist

def put_data(image,  left, right, curv_rad):
    font = cv2.FONT_HERSHEY_SIMPLEX
    text = 'Curve radius: ' + '{:04.2f}'.format(curv_rad) + 'm'
    cv2.putText(image, text, (40,70), font, 1.5, (200,255,155), 2, cv2.LINE_AA)
    direction = ''
    if curv_rad > 0:
        direction = 'right'
    else:
        direction = 'left'
    abs_curv_rad = abs(curv_rad)
    text = '{:04.3f}'.format(abs_curv_rad) + 'm ' + direction + ' of center'
    cv2.putText(image, text, (40,120), font, 1.5, (200,255,155), 2, cv2.LINE_AA)
    return image

for img in imgHolder:
    image = cv2.imread(img)
    result, left, right, curv_rad = imagePipeline(image)
    result = put_data(result, left, right, curv_rad)

    f, (axs1, axs2) = plt.subplots(1, 2, figsize=(15,11))
    axs1.set_title("Original image")
    axs1.imshow(image)
    axs2.set_title("Corner image")
    axs2.imshow(result)

print("Completed")

Completed

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.

class Line():
    def __init__(self):
        # was the line detected in the last iteration?
        self.detected = False  
        # x values of the last n fits of the line
        self.recent_xfitted = [] 
        #average x values of the fitted line over the last n iterations
        self.bestx = None     
        #polynomial coefficients averaged over the last n iterations
        self.best_fit = None  
        #polynomial coefficients for the most recent fit
        self.current_fit = [np.array([False])]  
        #radius of curvature of the line in some units
        self.radius_of_curvature = None 
        #distance in meters of vehicle center from the line
        self.line_base_pos = None 
        #difference in fit coefficients between last and new fits
        self.diffs = np.array([0,0,0], dtype='float') 
        #x values for detected line pixels
        self.allx = None  
        #y values for detected line pixels
        self.ally = None

def process_images(img):
    result, left, right, curv_rad = imagePipeline(img)
    output = put_data(result, left, right, curv_rad)
    return output

finalTest = cv2.imread('./test_images/test1.jpg')
finalResult = process_images(finalTest)
plt.imshow(finalResult)

<matplotlib.image.AxesImage at 0x12df934e0>

Provide a link to your final video output. Your pipeline should perform reasonably well on the entire project video (wobbly lines are ok but no catastrophic failures that would cause the car to drive off the road!).

Here is a link - https://youtu.be/oRey2z47zgk

from ipywidgets import interact, interactive, fixed
from moviepy.editor import VideoFileClip
from IPython.display import HTML
l_line = Line()
r_line = Line()

video_output1 = 'project_video_output.mp4'
video_input1 = VideoFileClip('project_video.mp4')
processed_video = video_input1.fl_image(process_images)
%time processed_video.write_videofile(video_output1, audio=False)

[MoviePy] >>>> Building video project_video_output.mp4
[MoviePy] Writing video project_video_output.mp4

100%|█████████▉| 1260/1261 [15:28<00:00,  1.30it/s]

[MoviePy] Done.
[MoviePy] >>>> Video ready: project_video_output.mp4 

CPU times: user 53min 13s, sys: 7min 15s, total: 1h 29s
Wall time: 15min 29s

video_output1 = 'harder_output.mp4'
video_input1 = VideoFileClip('harder_challenge_video.mp4')
processed_video = video_input1.fl_image(process_images)
%time processed_video.write_videofile(video_output1, audio=False)

[MoviePy] >>>> Building video harder_output.mp4
[MoviePy] Writing video harder_output.mp4

100%|█████████▉| 1199/1200 [21:03<00:00,  1.35it/s]   

[MoviePy] Done.
[MoviePy] >>>> Video ready: harder_output.mp4 

CPU times: user 51min 20s, sys: 6min 50s, total: 58min 11s
Wall time: 21min 4s

Briefly discuss any problems / issues you faced in your implementation of this project. Where will your pipeline likely fail? What could you do to make it more robust?

Solution - The above pipeline was able to detect the lane quite appropriately on the test video. But it faces some issues with the harder video input.

1. Was not able to correctly resolve the issue with thresholding, although this issue can be resolved with some experimentation.
2. Other issues is due to the image lighting issues, Not all the thresholds work properly on all the images.
3. This project has been fun to implement with different ways to implement and different ways to improve.
4. We can do some more operations on the images to enhance the lighting and other operations to improve the image so that it is better to detect the lane lines.